Updates July 14

PGrid

• Working on a new version of specs search ai which is still only specific to ssds but will allow searching for ssd specs based on a specs search queue which can be queued by classification ai
• Updated perplexity scraper to be able to switch between normal search and research modes
• WIP on SSD filters button and filters panel
• Added SSD scraping on amazon/ebay
• Updated SSD schema to remove a table I had purely for tracking SSD part numbers which would be joined to SSD variants. Refactored everything to use a json array in the SSD variants table.
• Updated generic ebay scraping to be more efficient by only only going into the listing page if the item is classified. Before it went into the details page for every single result.
• While doing ebay scraping updates, also fixed scattered logic for updating generic item price snapshots and item details to a single function called by both ebay and amazon search functions.
• Also while updating this generic scraping area, found that fail count wasn’t being updated for generic items. Fail count is used to relatively quickly remove listings that are no longer reachable from the frontend. This hasn’t been fixed yet
• Added infrastructure to do postgres integration tests. have basic tests just to see if the test postgres loads data correctly and has isolation but will be useful for future testing.
• Just in order to get the architecture correct before scaling up the number of categories and building up the classification and spec search system, started planning how to add sub variant of items so items will have the options of 3 levels deep of item specs data.